Fix OSE POD pagination truncating to first 2000 wells#116
Merged
Conversation
The OSE FeatureServer caps a page at maxRecordCount=2000, but chunk_size was 5000. The pagination loop breaks when a page returns fewer rows than chunk_size, so the first full 2000-row page read as "short" and the loop stopped after one request. Only the first 2000 PODs (OBJECTID-ordered = oldest first) were ever fetched statewide, so recent wells were dropped entirely -- leaving the POD-age products near-empty and under-counting every product fed by this source (well correlation, well density). Set chunk_size to 2000 so full pages read as full and the loop pages to the end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
nm_pod_age_by_county/nm_pod_age_pointscomputed near-empty (33 counties,total_recent_podsmax = 1; points collection had only 3 features), despite ~15,600 PODs in the source with afinish_datin the last 10 years.Root cause
The OSE
OSE_Points_of_DiversionFeatureServer caps a single page atmaxRecordCount = 2000, butNMOSEPODSiteSource.chunk_sizewas5000. The pagination loop stops when a page returns fewer rows thanchunk_size:So the first full 2000-row page read as "short" and the loop stopped after one request. Only the first 2000 PODs were ever fetched statewide — and because the server returns them in
OBJECTIDorder (oldest first), every recent well sat past row 2000 and was dropped. The POD-age products saw ~3 recent wells total.This silently under-counted every product fed by this source, not just POD-age: well correlation and county/basin well-density were all capped at the 2000 oldest PODs.
Fix
Set
chunk_size = 2000(≤ servermaxRecordCount) so full pages read as full and the loop pages to the end.Verification
maxRecordCount = 2000,supportsPagination = trueon the layer.finish_dat >= 2017-01-01.finish_dat=FINISH_DATE"Date well completed" (correct binning field);pod_status='ACT'(Active) and excluded basinsSP/SD/LWD(Surface Permit / Surface Declaration / Livestock Watering Declaration — not drilled wells) are semantically correct.Follow-up (not in this PR)
Re-run the affected assets to backfill:
nm_pod_age_by_county,nm_pod_age_points, well correlation, well density.🤖 Generated with Claude Code